Skip to content

Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality#11799

Open
landreev wants to merge 31 commits into
developfrom
11405-refactor-exports
Open

Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality#11799
landreev wants to merge 31 commits into
developfrom
11405-refactor-exports

Conversation

@landreev

@landreev landreev commented Sep 8, 2025

Copy link
Copy Markdown
Contributor

Un-drafting the PR now!

What this PR does / why we need it:

See issue #11405. Short version: the current export data framework is prohibitively expensive for datasets with very large amounts of tabular data (i.e., "ingested" DataFiles that have associated DataTable entities , with large numbers of child DataVariable objects). This PR improves it, via adding code that retrieves the data in manageable chunks. This in turn relies on the new version of the ExportDataProvider interface from https://github.com/gdcc/dataverse-spi.

Which issue(s) this PR closes:

Special notes for your reviewer:

Note that the PR builds with a snapshot release of dataverse-spi:

            <groupId>io.gdcc</groupId>
            <artifactId>dataverse-spi</artifactId>
            <version>2.1.0-SNAPSHOT</version>

I.e., the version 2.1.0 still needs to be properly released under gdcc. But I don't think that should be stopping putting the PR into review and merging it rather sooner than later.

Note that this PR also removes modules/dataverse-spi, and the action that used to build snapshot releases of it (.github/workflows/spi_release.yml) from the main source tree; now that dataverse-spi lives under gdcc.

Note specifically the test for the functionality, TabularDataExportIT added in this PR. I have added it to the list of integration tests that are performed automatically by the workflow action (in tests/integration-tests.txt) even though, strictly speaking, it is not a real integration test. I.e., it does not rely on calling a fully-functional Dataverse instance started remotely. It relies instead on the new test framework added in @poikilotherm's #12365 where a local Docker instance of the database is spun up in which some dummy objects - Datasets, Versions, Datatables and Variables etc are created instantly, allowing testing of complex functionality that requires real database access. Without this new framework, the only way to test an export of a dataset with N large tabular files would be to actually ingest these N files via the API in the remote Dataverse instance used by the RestAssured tests. Which would add to the ongoing bloating of the test suite, and still would not allow to test the underlying functionality as thoroughly (for example, there is no easy way to count the actual database queries under such scenario).

You can run the test in question in your own dev. environment via
mvn test -Dtest=TabularDataExportIT#exportTabularMetadata
If you do not have Docker running, the test will be politely skipped. Otherwise it will create a throwaway postgres instance and proceed to test tabular data exports and associated data retrieval, comparing the results produced using the legacy methods vs. the new and optimized versions added in this PR and more.

Suggestions on how to test this:

Please see the info under the "notes for the reviewer" and feel free to play and experiment with the new test described there. Please keep in mind that the test is still somewhat of a reference demonstrator of the new JpaEntityManagerService-based framework from #12365. It can be expanded and improved going forward.

The ultimate test would be to test before vs. after on some real-life monster datasets.
I would suggest these bad boys on qa:
https://qa.dataverse.org/dataset.xhtml?persistentId=doi:10.7910/DVN/KZW8R2 (207 ingested tab. files; 186906 variables total)
https://qa.dataverse.org/dataset.xhtml?persistentId=doi:10.7910/DVN/S5BRBO (134 tab. files; 536006 variables).

"ddi" (aka the "full" ddi) is the export format that needs to be tested.
Note that some extra steps will be required in order to test this on qa:

  • You will want to change the storage driver to local filesystem ("file") for the datasets in question - so that Dataverse can write the export files;
  • And make sure you are testing the export code vs. looking at a potentially existing cached export; i.e., the file export_ddi.cached will need to be erased between testing runs.
    Happy to assist with the above if needed.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

@landreev landreev self-assigned this Sep 8, 2025
@github-actions github-actions Bot added FY25 Sprint 24 FY25 Sprint 24 (2025-05-21 - 2025-06-04) FY25 Sprint 25 FY25 Sprint 25 (2025-06-04 - 2025-06-18) FY25 Sprint 26 FY25 Sprint 26 (2025-06-18 - 2025-07-02) FY26 Sprint 1 FY26 Sprint 1 (2025-07-02 - 2025-07-16) FY26 Sprint 2 FY26 Sprint 2 (2025-07-16 - 2025-07-30) FY26 Sprint 3 (2025-07-30 - 2025-08-13) FY26 Sprint 4 FY26 Sprint 4 (2025-08-13 - 2025-08-27) FY26 Sprint 5 FY26 Sprint 5 (2025-08-27 - 2025-09-10) Size: 80 A percentage of a sprint. 56 hours. labels Sep 8, 2025
@github-actions

This comment has been minimized.

@landreev

Copy link
Copy Markdown
Contributor Author

Note that the PR now builds under Jenkins (with a snapshot release of dataverse-spi that is now published).

@github-actions

This comment has been minimized.

@cmbz cmbz added the FY26 Sprint 24 FY26 Sprint 24 (2026-05-20 - 2026-06-03) label May 21, 2026
@cmbz cmbz added the FY26 Sprint 25 FY26 Sprint 25 (2026-06-03 - 2026-06-17) label Jun 3, 2026
landreev added 2 commits June 18, 2026 13:57
Resolved the following merge conflicts:
	modules/dataverse-parent/pom.xml
	src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java
	src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java
@cmbz cmbz added the FY26 Sprint 26 FY26 Sprint 26 (2026-06-17 - 2026-07-01) label Jun 18, 2026
@cmbz cmbz added the FY27 Sprint 1 FY27 Sprint 1 (2026-07-01 - 2026-07-15) label Jul 1, 2026
@cmbz cmbz added the FY27 Sprint 2 FY27 Sprint 2 (2026-07-15 - 2026-07-29) label Jul 15, 2026
landreev added 5 commits July 19, 2026 22:39
... into IQSS/11405-refactor-exports (the working branch behind an already existing PR). 
merging without further ado.
…ing as is when committed, since there was only one dataset and one datasetversion in the throwaway database. #11405
…ularly-scheduled workflow action integration tests. #11405
Resolved conflicts:
	src/main/java/edu/harvard/iq/dataverse/api/Metadata.java
(seemingly trivial conflict, on account of a new API parameter added in this PR vs. extra annotations having been added in the develop branch). #11405
assertTrue(false, "Unexpected XMLStreamException when attempting to parse the dataDscr section: " + xse.getMessage());
} finally {
if (xmlr != null) {
try {xmlr.close();} catch (XMLStreamException e) {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [reviewdog] <com.puppycrawl.tools.checkstyle.checks.blocks.LeftCurlyCheck> reported by reviewdog 🐶
'{' at column 21 should have line break after.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't tell me what to do, reviewdog. That was formatted like that on purpose, to emphasize the fact that we don't care about that particular exception at that stage of the test.

@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown

Test Results

404 tests  +1   389 ✅ +1   38m 30s ⏱️ + 13m 12s
 56 suites +1    15 💤 ±0 
 56 files   +1     0 ❌ ±0 

Results for commit 3d318be. ± Comparison against base commit df71344.

♻️ This comment has been updated with latest results.

@landreev
landreev marked this pull request as ready for review July 20, 2026 16:34
@landreev landreev moved this from In Progress 💻 to Ready for Review ⏩ in IQSS Dataverse Project Jul 20, 2026
@landreev landreev removed their assignment Jul 20, 2026
@landreev

landreev commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@poikilotherm Hi, I'm looking at the test results above, and seeing this failure to "Release Snapshot" for Dataverse SPI:

Screenshot 2026-07-20 at 1 55 13 PM

Am I correct assuming this is an old action that simply needs to be removed, now that dataverse-spi lives in https://github.com/gdcc/dataverse-spi?
Similarly, should I go ahead and purge modules/dataverse-spi from the main source tree?
Please stop me if there is any reasons why I should not delete these.
Thank you!

@poikilotherm

Copy link
Copy Markdown
Contributor

@landreev Yes, the in-tree SPI stuff can be completely eradicated. ☢️ 😄

landreev added 2 commits July 21, 2026 14:47
…hot releases of it from the source tree; since dataverse-spi has been living under https://github.com/gdcc/dataverse-spi for some time now. #11405
@github-actions

Copy link
Copy Markdown

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:11405-refactor-exports
ghcr.io/gdcc/configbaker:11405-refactor-exports

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

@poikilotherm
poikilotherm self-requested a review July 22, 2026 11:30
// DO in fact contain enough information to generate the otherMat sections
// properly, whether this is a short or a full version of the DDI. I am however leaving
// this method here for reference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per your comment, IMO this method should be tagged @deprecated to indicate its imminent removal.

* @varQuantityMap mapping of ordered datatables -> number of variables in each
* @exportDataProvider data provider instance
* @variablesBatchSize number of variables to use in forming processing batches
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integration test should be in the same package as the class (it already is in ...export, so it should move to ...export.ddi, the package this class is a part of). Then the method can be made package private and is accessible to the test right away without exposing it.

logger.fine("total number of variables in this batch: " + varQuantityThisBatch);

int count = 0;
//for (int i = 0; i < tabularFileDetails.size(); i++) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a comment to explain the iterator pattern? If so, please use human language and remove the code block ;-)


}

public void exportFormats(Dataset dataset, List<String> formatNames) throws ExportException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is missing Javadocs explaining scope, reasoning and behavior.

Also:

  1. I encourage you to use an empty list instead of a null value, which IMO is much more Javaish.
  2. There is no null check on the dataset parameter.
  3. Why not use DatasetVersion and split the logic in two methods? The getExport() method uses a DatasetVersion, thus IMO using a DatasetVersion + list of formats is more aligned. There can be a second method with Dataset + list of formats for convenience, that handles the business logic of which version may be exported (calling the one with version + formats internally).

For Johannes I will be adding a method with cardinality in the other direction: one format, multiple versions. Eventually, we will refactor the service to support exporting arbitrary versions, where it also makes sense to have version + formats.

Set<DatasetMetadataPredicates> predicates = query.getDatasetPredicates();

for (DatasetMetadataPredicates p : predicates) {
// @todo This is pending on adding a dedicated DATASET_LEVEL_ONLY predicate

@poikilotherm poikilotherm Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The DatasetExportQuery has predicates only for the dataset's own metadata:

Predicates for controlling which dataset metadata is included in an export and at what level of detail.

The dataset export query may have a distinct FileExportQuery containing detailed information as its own predicates about the file metadata. At the moment, there is only one case where this method may return false:

  1. There is a file export query within the dataset export query and it has a predicate "SKIP_FILES". (Otherwise it's either all files, public and/or tabular only).

We might need to tighten the definition in the export query spec, here's what it says now:

File metadata shaping is optional: if no FileExportQuery is provided, methods that include file metadata will apply their own defaults. Methods that do not return file metadata will ignore any nested FileExportQuery.

At time of writing, this seemed fine. But maybe it isn't? Should we do an explicit default? Do I remember correctly that in the past getDatasetJson() did not return file level metadata? (Then we should keep this default for backward compatibility, changing it would require a breaking change - and we shouldn't do those at this point.)

// generated, should we return it (potentially moving MUCH more json
// than the client needs, or spend extra cycles generating the short
// form from scratch? - I'm choosing to go with latter.
if (jsonRepresentationNoFiles == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My gut tells me: if we really want to cache here, let's use a thread-safe map and the query as the key. In the edge case that someone really wants to retrieve the metadata again, they get a consistent result, as the query might be different the next time around!

dataset.setLastExportTime(null);
}

public void clearCachedFormats(Dataset dataset, List<String> formatNames) throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behavior here is not aligned with exportFormats. With that method using a null list will clear all, here you don't even check for null, risking an NPE.

I'd argue the same way as exportFormats: use a List, either empty or filled, disallow null. If empty, clear all.

}

dataset.setLastExportTime(null);
public void clearAllCachedFormats(Dataset dataset) throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a redundant method. People can just call the other defined methods. If you really want this for convenience, don't add a second implementation.

@@ -1 +1 @@
DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,HarvestingClientsIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT,InvalidCharactersIT,LicensesIT,NotificationsIT,BagIT,MetadataBlocksIT,NetcdfIT,SignpostingIT,FitsIT,LogoutIT,DataRetrieverApiIT,ProvIT,S3AccessIT,OpenApiIT,InfoIT,DatasetFieldsIT,SavedSearchIT,DatasetTypesIT,DataverseFeaturedItemsIT,SendFeedbackApiIT,CustomizationIT,JsonLDExportIT,WorkflowsIT,LDNInboxIT,LocalContextsIT,CorsIT,GuestbooksIT,LocallyFairIT
DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,HarvestingClientsIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT,InvalidCharactersIT,LicensesIT,NotificationsIT,BagIT,MetadataBlocksIT,NetcdfIT,SignpostingIT,FitsIT,LogoutIT,DataRetrieverApiIT,ProvIT,S3AccessIT,OpenApiIT,InfoIT,DatasetFieldsIT,SavedSearchIT,DatasetTypesIT,DataverseFeaturedItemsIT,SendFeedbackApiIT,CustomizationIT,JsonLDExportIT,WorkflowsIT,LDNInboxIT,LocalContextsIT,CorsIT,GuestbooksIT,LocallyFairIT,TabularDataExportIT

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not add this test here. It's an integration test, not an API test. For historic reasons, this file has been around and collects these API tests. It should be torched with fire.

Your performance test will be automatically picked up for execution during mvn verify -Dit.groups=performance

If you feel we should make performance tests a part of the regular Github Action flow, let's add it there! (currently only integration and migration are picked up, see pom.xml property it.groups)

datasetService.reExportDatasetAsync(dataset);
List<String> formatNames = null;
if (formats != null) {
formatNames = new ArrayList<>(Arrays.asList(formats.split(",")));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As user input, the list of formats shall be verified on this code level, before entering any services.

Side note: The try/catch thing around everything is an anti-pattern and always has been. Let's not fix it here, but I wanted to repeat this, as it is an anti pattern all over our API code.

@poikilotherm

poikilotherm commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Sorry, I submitted the review before adding a comment! My bad!

Overall, I think the SPI refactoring was a good choice. The code here is much clearer to read now, as responsibilities are clearer.
There are still a few kinks, which should be addressed. I'd be happy to help if you want to do a coding session via VC. Some of my remarks might be best solved by talking about it and improving wording and code on the SPI end. Let me know if you'd prefer me just sending patches... 🙈

@landreev landreev moved this from Ready for Review ⏩ to In Review 🔎 in IQSS Dataverse Project Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

FY25 Sprint 24 FY25 Sprint 24 (2025-05-21 - 2025-06-04) FY25 Sprint 25 FY25 Sprint 25 (2025-06-04 - 2025-06-18) FY25 Sprint 26 FY25 Sprint 26 (2025-06-18 - 2025-07-02) FY26 Sprint 1 FY26 Sprint 1 (2025-07-02 - 2025-07-16) FY26 Sprint 2 FY26 Sprint 2 (2025-07-16 - 2025-07-30) FY26 Sprint 3 (2025-07-30 - 2025-08-13) FY26 Sprint 4 FY26 Sprint 4 (2025-08-13 - 2025-08-27) FY26 Sprint 5 FY26 Sprint 5 (2025-08-27 - 2025-09-10) FY26 Sprint 9 FY26 Sprint 9 (2025-10-22 - 2025-11-05) FY26 Sprint 10 FY26 Sprint 10 (2025-11-05 - 2025-11-19) FY26 Sprint 11 FY26 Sprint 11 (2025-11-20 - 2025-12-03) FY26 Sprint 12 FY26 Sprint 12 (2025-12-03 - 2025-12-17) FY26 Sprint 13 FY26 Sprint 13 (2025-12-17 - 2025-12-31) FY26 Sprint 14 FY26 Sprint 14 (2025-12-31 - 2026-01-14) FY26 Sprint 15 FY26 Sprint 15 (2026-01-14 - 2026-01-28) FY26 Sprint 16 FY26 Sprint 16 (2026-01-28 - 2026-02-11) FY26 Sprint 17 FY26 Sprint 17 (2026-02-11 - 2026-02-25) FY26 Sprint 22 FY26 Sprint 22 (2026-04-22 - 2026-05-06) FY26 Sprint 23 FY26 Sprint 23 (2026-05-06 - 2026-05-20) FY26 Sprint 24 FY26 Sprint 24 (2026-05-20 - 2026-06-03) FY26 Sprint 25 FY26 Sprint 25 (2026-06-03 - 2026-06-17) FY26 Sprint 26 FY26 Sprint 26 (2026-06-17 - 2026-07-01) FY27 Sprint 1 FY27 Sprint 1 (2026-07-01 - 2026-07-15) FY27 Sprint 2 FY27 Sprint 2 (2026-07-15 - 2026-07-29) Size: 80 A percentage of a sprint. 56 hours.

Projects

Status: In Review 🔎

Development

Successfully merging this pull request may close these issues.

Refactor and optimize the Export metadata framework, especially for the data/variable-level metadata

4 participants